1. Field of the Invention
The present invention relates to a neural network representing apparatus, and more particularly, it relates to a neural network representing apparatus having a self-organizing function. More specifically, the present invention relates to a synapse load correction scheme in self organization of a neural network representing apparatus.
2. Description of the Background Art
Neural networks have been widely implemented by various electronic circuits on the model of vital cells. Such a neural network representing apparatus has a self-organizing function, i.e., a learning function of correcting internal synapse load values in accordance with externally supplied educator information. "A Neuromorphic VLSI Learning System" by J. Alspector et al., in "Advanced Research in VLSI 1987", issued by MIT Press, pp. 313 to 327 discloses an exemplary neural network representing apparatus. The structure and operation of such a conventional neural network representing apparatus are now described.
FIG. 1 shows the structure of an integrated neural network which is disclosed in the above literature. Referring to FIG. 1, the conventional neural network includes complementary data input line pairs IN1 and IN1 to INj and INj, which are arrayed in the row direction and complementary data output line pairs S1 and S1 to Sj and Sj, which are arrayed in the column direction. This neural network further comprises differential amplifiers Zk which detect/amplify potential differences on the data input line pairs INk and INk (k=1 to j) and transfer the same onto the data output lines Sk and Sk. Such differential amplifiers Z1 to Zj are arranged on a diagonal line of a connection matrix (matrix formed by the data input lines, the data output lines and resistive elements).
The respective differential amplifiers Zk (k=1 to j) correspond to neurons, while the data input lines IN and IN correspond to dendrites, and the data output lines S and S correspond to axons. In the following description, symbols IN and S generically denote the data input lines and the data output lines respectively. Synapse loads W are provided by resistive elements R which are arranged on crossings between the data input lines IN and IN and the data output lines S and S.
The differential amplifier Zk has complementary outputs S and S. When the neuron is in an "ON state", the output of the differential amplifier Zk goes to "1" (5 V), while the output S goes to "0" (0 V) when the same is in an OFF state. The outputs of such differential amplifiers Zk are fed back to the data input lines IN and IN through resistive elements R, which are arranged in a matrix to represent synapse loads.
A resistive element R which is arranged on an i-th row on a j-th column of the connection matrix connects the output of the differential amplifier (neuron) Zj to the input of the differential amplifier (neuron) Zi. When a synapse load Wij is positive, the data output line Sj is connected to the data input line INi and the complementary data output line Sj is connected to the complementary data input line INi. When a synapse load Wij is negative, the complementary data output line Sj is connected to the data input line INi, and the data output line Sj is connected to the complementary data input line INi.
A differential amplifier Zt which is provided in a region V of the connection matrix is regularly in an "ON state", and an output line Sv is regularly supplied with a signal of "1" and a complementary output line Sv is regularly supplied with a signal of "0". Such a differential amplifier Zt serves to eliminate influences by threshold values in the respective neuron units, thereby equivalently setting the threshold values of the respective neuron units at zero.
This neural network is initialized by setting the weight (resistance value) of each resistive element R. Data of the synapse load Wij can be transferred along bold arrows shown in FIG. 1 through a weight processing circuit which is provided in correspondence to each resistive element R, as hereinafter described.
FIG. 2 shows the structure of each synapse load block (resistive element). The synapse load block includes four transistor groups TR1, TR2, TR3 and TR4, in order to provide positive coupling (excitatory coupling) and negative coupling (inhibitory coupling). Each of the transistor groups TR1 to TR4, which are identical in structure to each other, includes n MIS (metal-insulating film-semiconductor) transistors T0 to Tn-1 and one pass transistor TG. ON-resistances of the MIS transistors T0 to Tn-1 are set at 1:2: . . . : 2.sup.n-1, in order to provide different resistance values.
The pass transistors TG1 and TG4 receive a signal TSGN showing the sign of the synapse value in the gates thereof, while the pass transistors TG2 and TG3 receive a complementary signal TSGN at the gates thereof. The signals TSGN and TSGN are complementary to each other, and positiveness/negativeness of the sign of the synapse load is determined by the signals. When the synapse load Wij is positive, the signal TSGN goes to "1", and the synapse load Wij is provided by the transistor groups TR1 and TR4. When the synapse load Wij is negative, the complementary signal TSGN goes to "1", and the synapse load Wij is provided by the transistor groups TR2 and TR3.
The synapse load Wij is set by bringing one or more MIS transistors in each transistor group TR into ON states by an output from a weight processing circuit. The structure and operation of the weight processing circuit for setting the synapse load are now described.
FIG. 3 illustrates the structure of the weight processing circuit for correcting the synapse load in self organization. This weight processing circuit is provided for each synapse load Wij, and includes a correlation logic CL, an up/down logic UDL, and flip-flops FF0 to FFn.
The flip-flops FF0 to FFn control ON/OFF states of the MIS transistors for representing the corresponding synapse load Wij. The flip-flop FF0 stores information indicating the sign of the synapse load Wij, and controls ON/OFF operations of the pass transistors TG. The flip-flops FF0 to FFn control ON/OFF states of the MIS transistors T0 to Tn-1 (see FIG. 2).
The correlation logic CL receives a signal "Phase" indicating the phase in the operation and a signal "COOC" indicating the times when both outputs Si and Sj of the neuron units (differential amplifiers) Zi and Zj are "1", and counts the number thereof, thereby evaluating a probability distribution (expected value) Pij. This correlation logic CL receives a weight adjusting signal ADW and supplies a signal indicating an increment, a decrement or holding (silent state) to the up/down logic UDL, in accordance with the following equation through the evaluated probability distribution Pij: EQU .DELTA.Wij=.eta..multidot.(P.sup.+ ij-P.sup.- ij)
where .eta.represents a positive constant called a learning coefficient, which defines the amount of correction of the synapse load in a single learning operation. Signs + and - added to Pij indicate that these are probability distributions which are obtained in plus and minus phases respectively. In the plus phase, input neurons and output neurons are champed to educator information data (data pattern to be learned). In the minus phase, only the input neurons are clamped at the educator information.
The up/down logic UDL increments, decrements or holds its count value in response to an increment/decrement instruction signal received from the correlation logic CL, and transfers the same to the flip-flops FF0 to FFn. The up/down logic UDL, having the structure of a shift register, can receive a synapse load W from an up/down logic which is included in an adjacent weight processing circuit and transfer the data to another up/down logic which is included in an adjacent subsequent stage in initialization.
FIG. 4 illustrates an exemplary structure of the up/down logic UDL. In the structure shown in FIG. 4, each synapse load W is displayed in four bits (including one sign bit), for example. FIG. 4 illustrates no path for setting weight data from an adjacent weight processing circuit shown in FIG. 3.
The up/down logic UDL is formed by an up/down counter 100. The up/down counter 100 comprises a terminal U/D which receives a signal instructing increment/decrement of a count value, a terminal T which receives a signal providing a change timing for the count value, a reset terminal R, and data output terminals Q0 to Q3. Outputs from the output terminals Q0 to Q2 provide the value of the synapse load W, while the output terminal Q3 outputs data defining the sign of the synapse load W. The output data from the output terminal Q3 is transferred through an inverter Il. The output terminals Q0 to Q3 are coupled to flip-flops FF0 to FFn (n=3) through signal lines 103 to 106.
In accordance with an increment/decrement instruction signal transferred through a signal line 102, the up/down counter 100 increments, decrements or holds its count value in response to a timing signal which is transferred through a signal line 101. Learning of the synapse load is made by this operation.
A threshold processing operation of one neuron (differential amplifier) Zi is now described with reference to FIG. 5. First, the structure of the differential amplifier Zi is described.
Referring to FIG. 5, the differential amplifier (neuron i) Zi comprises two pairs of differential inputs and a pair of differential outputs. The first differential input pair differentially amplifies the potential difference on data input lines IN and IN, and provides an energy gap ##EQU1## The second differential input pair receives a complementary output from a differential amplifier AZ for generating noises, and generates an annealing temperature T in the form of a noise.
The amplifier AZ receives a noise signal from a noise source NS. The noise signal from the amplifier AZ is so set that its level is reduced with progress of a certain phase. Thus, implemented is such a process that annealing is started at a high temperature level and the annealing temperature is successively reduced so that the neural network is stabilized at the global minimum without capture in pseudo optimum solutions (local minima). The amplifier AZ is generally formed by an operational amplifier, whose gain is adjusted from an externally provided device in order to set the annealing temperature.
The threshold processing operation of the differential amplifier Zi is now described.
Referring to FIG. 5, one transistor TC represents the conductance of each synapse load W. When a synapse load W is positive and a neuron corresponding thereto is in an "ON state" (differential amplifier output S is "1") or the synapse load W is negative and the neuron related thereto is in an "OFF state", voltages Von and Voff are transferred to the data input lines IN and IN through conductances of I and IV rows. Conductances of the II and III rows are those in such a case that the synapse load W is negative and the neuron is in an "ON state" or the synapse load W is positive and the neuron is in an "OFF state". In this case, the voltage Voff is transferred to the data input line IN, and the voltage Von is transferred to the complementary data input line IN.
The positive input of the differential amplifier Zi is coupled with a conductance for pulling up the same to the voltage Von and that for pulling down the same to the voltage Voff. The conductance for pulling up the positive input of the differential amplifier Zi to the voltage Von is provided by the absolute value of the sum of positive synapse loads from "ON state" neurons and negative synapse loads W from "OFF state" neurons. The conductance for pulling down the potential of the positive input of the differential amplifier Zi to the voltage Voff is provided by the absolute value of the total sum of negative synapse loads from "ON state" neurons and positive synapse loads W from "OFF state" neurons. Relation of the conductances at the negative input of the differential amplifier Zi is inverse to that at the positive input of the differential amplifier Zi.
Considering the aforementioned relation as well as the fact that the synapse load in the region V shown in FIG. 1 is expressed as -.theta.i, the differential amplifier Zi simply performs the following comparison: ##EQU2## where Sj*=1:Sj=1
Sj*=-1:Sj=0
The differential amplifier Zi performs threshold processing in accordance with the aforementioned expression, and outputs data (state signals) to data output lines S and S.
In this structure, the value of the positive input of the differential amplifier Zi may be simply compared with a threshold value (Von+Voff)/2, thereby obtaining desired output data.
There has also been proposed a structure of a synapse load expressing circuit which stores information indicating a synapse load in a register and transfers a current from a constant current source onto a dendrite signal line in response to "1" or "0" of a signal Sj on an axon signal line and the information stored in the register. An exemplary structure using the constant current source is described in pages 10.1.1 to 10.1.7 of an article entitled "Electronic Circuit Implementation of Neuron Morphic System" by Jack I. Ruffel, IEEE 1988 "Custom Integrated Circuit Conference". Also in this synapse load expressing circuit, conductance ratios of MIS transistors are so adjusted as to transfer a desired product signal Wij.multidot.Sj onto the dendrite signal line.
In a conventional neural network representing apparatus, synapse loads are corrected in accordance with the following expression: EQU .DELTA.Wij=.eta..multidot.(p.sup.+ ij-p.sup.- ij)
The synapse load correction coefficient (learning coefficient) .eta. is a positive constant, which is set at a common value to all synapse load expressing circuits. This means that overall neurons are identical in self-organizing efficiency to each other, which is equivalent to such interpretation that an influence exerted by a remote neuron to a certain neuron is identical to that exerted by a neuron which is close to this neuron.
In the vital brain, it may be predicted that interaction between neurons is reduced as the distance therebetween is increased, due to increase in propagation time of an axon signal and attenuation of the signal in propagation. In other words, it can be said that correction of synapse loads is greatly influenced by spatial positional relations, such as interdistances, between related neurons.
Thus, the self-organizing model of the conventional neural network device is too much simplified, and it has been impossible to perform self organization precisely reflecting the function of the vital brain, i.e., the learning process of an organism.